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Background / Context: 

Summer school is frequently used as a policy lever to improve academic outcomes for 
low-achieving students and low-income students (Jacob & Lefgren, 2004; Mariano & Martorell, 
2013; Marsh, Gershwin, Kirby, & Xia, 2009; Matsudaira, 2008; McCombs, Augustine, 

Schwartz, Bodilly, Mclnnis, Lichter, & Cross, 2011). One reason academic summer programs 
are proposed for students from low income backgrounds is that disadvantaged students fall 
behind their higher income peers academically over the summer vacation months (e.g. Burkam, 
Ready, Lee, & LoGerfo, 2004; Cooper et al., 1996; Downey, von Hippel, & Broh, 2004). 
Although effective summer math and reading interventions have the potential to prevent summer 
slide for low income students (Cooper et al., 2000), there is also concern that compensatory 
interventions widen, rather than narrow, income -based test score gaps because privileged 
students sometimes benefit more from such programs than do underprivileged students (Ceci & 
Papiemo, 2005). In their meta-analytic review, Cooper et al. (2000) found that this may in fact 
be the case for academic summer interventions; their evidence showed that higher income 
students benefitted more from summer school than did low income students. However, in our 
recent meta- analysis (Kim & Quinn, 2013) of studies published after Cooper et al.’s (2000) 
review, we found the opposite: summer reading programs were more effective for low income 
students than for higher income students. 

One possible explanation for why Cooper et al.’s (2000) result differed from ours (Kim & 
Quinn, 2013) relates to the academic subject matter: Cooper et al. (2000) combined both math 
and reading outcomes in their meta-analysis, while we (Kim & Quinn, 2013) analyzed only 
reading outcomes. Different patterns of summer learning in math and reading support different 
predictions about the moderation of summer program effects by student income status. Using 
relative measures of change, middle class students make more progress in reading over the 
summer than do low income students (c/=0.06 for middle income students compared to d=-0.2\ 
for low income students according to Cooper et al., 1996). For math, however, the amount of 
relative summer change does not differ by income status (Cooper et al., 1996). In a summer 
school study, then, posttest treatment-control contrasts in reading may be greater for low income 
students even if summer schools promote the same spring-to-fall reading gain, on average, for 
students of all income groups (Kim & Quinn, 2013). The same may not be true for math. It is 
therefore important to test whether the moderation we observed for recent summer reading 
studies replicates for recent math studies. 

A related question is whether summer school has similar effects on math and reading 
outcomes in general. Descriptively, Cooper et al. (2000) found that math effects tended to be 
slightly larger than reading effects; however, the magnitude and statistical significance of this 
difference was sensitive to model specifications (with effects ranging from d=0.22 to 0.26 for 
reading and from 0.26 to 0.30 for math). In another review of the out-of-school time literature, 
Lauer and colleagues (Lauer, Akia, Wilkerson, Apthorp, Snow, & Martin-Glenn, 2006) found 
slightly smaller summer school effects on reading than math (<:/=(). 05 for reading compared to 
d=0.09 for math), though they did not conduct a formal hypothesis test on this difference (nor 
did they test the moderating role of student income status). 

Purpose / Objective / Research Question / Focus of Study: 

The finding that academic summer programs are effective for low income students has 
replicated across meta-analytic reviews. However, these reviews have yielded contradictory 
evidence about whether summer programs are more effective for lower- or higher-income 
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students. This discrepancy may be due to income -based differences in the summer 
counterfactual that exist for reading but not for math. Additionally, the related question of 
whether the effects of summer school differ for math and for reading in general has not yielded 
robust findings to date and bears replication with updated studies. In the present study, we 
therefore extend our previous research on summer learning programs in two ways: 1) We 
conduct a comprehensive meta-analysis of the literature on math summer school programs (from 
1998 to 2011), and test whether the moderating role of income that we observed in reading over 
this period replicates for math; 2) We ask whether the effect of summer programs on student 
outcomes differs for math compared to reading. 

Research Design: 

Literature Search Procedures, Selection Criteria, Coding, and Analysis 

We searched the literature through three primary channels: (1) online academic research 
databases (Academic Search Premier, Education Abstracts, ERIC, PsycINFO, EconLit, ProQuest 
Dissertations and Theses) and targeted internet sites (Child Trends LINKS, What Works 
Clearinghouse, Harvard Family Research Project’s Out-of-School Time Database, MDRC, 
NBER, RAND, Mathematica, SEDL, Wallace Foundation); (2) reference lists of previous review 
articles (Bodilly & Beckett, 2005; Lauer et al., 2006; McCombs et al. 2011; Terzian, Moore, & 
Hamilton, 2009); and (3) soliciting research reports from government education agencies. We 
conducted our search in June 201 1 for studies that were released after August 1998, which was 
the final month included in Cooper et al.’s (2000) review. 

The studies included in our review met 5 selection criteria. We required that studies: (1) 
evaluate the effects of a classroom- or home -based summer reading or school-based math 
intervention in the United States or Canada, (2) administer a standardized math or reading test, 

(3) provide sufficient empirical information to compute an effect size (Cohen’s d), (4) include 
students who were in Kindergarten to Grade 8 prior to enrollment in a summer math or reading 
intervention, and (5) compare the performance of students in a treatment group to the 
performance of students in a control group who did not participate in the treatment or 
systematically receive an alternative intervention. We searched the electronic data bases with the 
following key words: “summer program*,” “summer school*,” “summer reading,” “summer 
literacy,” “summer enrichment,” “summer remedia*,” “summer instruction*,” “summer 
education*,” “summer learning”, “math*,” “*algebra*,” “STEM,” "*experiment*," "control*," 
“regression discontinuity,” "compared," "comparison," “field trial*,” "effect size,” "evaluation”. 

We included reading studies that administered a variety of literacy assessments. In cases 
where a study tested multiple reading skills (e.g. vocabulary, comprehension, and fluency), we 
combined the results from each reading assessment into a single “reading total” effect size (none 
of the math studies reported separate effect sizes for different math sub-domains). When results 
were reported separately for grade-level subgroups within a study, we took the mean effect size 
across grade levels. 

To conduct the income status moderator analyses, we coded the study samples based on 
the percent of participating students who were low income, or free/reduced-price lunch (FRL) 
eligible. Samples that included over 50% FRL students were coded as “low income”; samples 
with less than 50% were coded as “mixed income” samples. 

To synthesize the results of these studies, we use random effects meta-analysis. 
Specifically, we fit the model: 

dj = Yo + YsW sj + fij + £j , 
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where dj represents the effect size for study j, 1/14^ represents an indicator for whether the sample 
was low- or mixed-income, fij represents the study-level error, and represents the error due to 
sampling of students within study. 

Findings / Results: 

The Moderating Role of Income Status 

Our search resulted in 35 studies of summer reading programs, which reported the effects of 
41 different interventions, and 12 studies of summer math programs. 

In Table 1, we present the meta- analytic results for the math studies, both overall and 
separately by student income status. Overall, summer math programs have a significant positive 
effect on students’ math scores (d=0.08). However, the pattern of effects differs by income 
status. In low income samples, students enjoyed a significant positive treatment effect of 
<r/=0.11, while in mixed income samples, the pooled effect size was negatively-signed and not 
statistically different from zero (d=- 0.02). Due to the small number of studies, however, we 
have limited statistical power for detecting whether effects were statistically different for each 
group (p=0.09). 

dnsert Table 1 about Hero 

Within-Study Analyses 

In Table 2, we present the results of within-study analyses (using fixed effect meta-analytic 
models). Because some programs included both math and reading components, we were able to 
conduct a within-study moderator analysis addressing the question of whether the pooled effects 
on math and reading were the same, after controlling for program. These results indicate that 
program effects on math and reading were similar. In Figure 1, we demonstrate the similarity in 
studies’ math and reading effects graphically through a scatter plot of math effect sizes on 
reading effect sizes (studies’ bubbles are weighted to reflect the inverse of the variance of the 
math effect size). Effect sizes are highly correlated, with an unweighted Pearson’s r of 0.93 
(weighted r=0.98; Spearman’s rho= 0.94). 

dnsert Table 2 about Hero 
dnsert Figure 1 about Hero 

We also conduct within-study moderator analyses for student income status to test the 
sensitivity of our results from Table 1 to model specification. For the subset of math studies that 
reported effects separately by student income status, we found that low income students 
benefitted more from summer math programs than did higher income students attending the 
same programs, and this difference was statistically significant. This allows us to rule out the 
possibility that the moderator result from Table 1 was due to cross-study confounders, and 
replicates the within-study income moderation analysis for reading (the results of which we also 
include in Table 2 for reference). The overall pattern of results, then, suggests that low income 
students benefit more from academic summer programs than do higher income students, in both 
math and reading. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

In this study, we examined whether the moderating effect of student income status observed 
in a meta-analysis of summer reading programs replicated for math. We found evidence that the 
result does replicate for math, with stronger effects for low income students than for higher- 
income students. We also found that the effects of summer programs did not differ for math and 
reading outcomes; on the contrary, programs with both math and reading components 
demonstrated effect sizes that were highly correlated between content areas. 
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One hypothesis as to why our moderator result differed from Cooper et al.’s (2000) is that the 
summer counterfactual for the average higher income student may have changed from the period 
of Cooper’s review (which included studies from 1966 to 1998) to the period of our review 
(1998 to 201 1). Today, the achievement returns to family income are 20-50% larger than they 
were for children born in the 1970s, and this may be due to parents’ increased levels of 
investment in their children’s cognitive development (Reardon, 2011). If academic investment 
by wealthier parents is higher today than 40 years ago, the summer counterfactual for higher 
income students may also be quite different today compared to the past. Given that the effects of 
summer school depend in part on the amount of loss or gain experienced by nonparticipants, the 
summer school treatment/control contrast for higher income students may be weaker today than 
in the past. 

Of course, there is always the possibility that the discrepant findings are due simply to 
chance, particularly when sample sizes (and in the case of random effects meta-analysis, number 
of studies) are small (Allison, 1999). The middle income subgroup in Cooper et al.’s (2000) 
moderator analysis consisted of effect sizes from only four studies, while our math analysis had 
only 3 middle-income samples. Furthermore, despite the overall discrepancy in results across 
reviews, similarities exist. One of the studies from Cooper et al.’s (2000) middle income 
subgroup reported separate effects for Title 1 students, migrant students, and middle class 
students; while effects were negative for Title 1 and middle class students, effects were positive 
for migrant students, who “probably represent students who suffer the greatest economic 
disadvantage” (Cooper et ah, 2000, p. 63). In fact, other studies in Cooper et al.’s (2000) meta- 
analysis featuring children of migrant workers exhibited some of the largest effect sizes in the 
review. This is consistent with our result that low income students benefit more from summer 
programs than do higher income students. 

The limitations of the current study call attention to areas in which more research is needed. 
To begin with, the number of studies on the effects of math summer school is low. We were able 
to find over three times as many studies of summer reading programs in our review window. 

The math studies that we did find often did not report enough information that would allow us to 
explore the impacts of theoretically important potential moderators. If, as prior research 
suggests, higher income students have a summer advantage in reading but not math, then there is 
not an easy explanation for why we observed the moderating effect of income for both math and 
reading outcomes. Unfortunately, few math studies reported the extent of summer learning loss 
in the control group, which prevented us from examining the role of summer loss in explaining 
the effect size patterns of our included studies. Additionally, none of the studies disaggregated 
the math results by sub-domain. Previous research suggests that students experience large losses 
in math computation over the summer while making gains in problem-solving (Cooper et al., 
1996). The effects of math summer school may therefore differ by the math skills that are taught 
and/or assessed, and cross-study differences in this area may explain cross-study differences in 
effect sizes. Unfortunately, we are unable to test this hypothesis. We were also unable to probe 
the reasons behind the high correlation between math and reading effects due to generally limited 
descriptions of the actual instruction provided by the programs. While our investigation provides 
additional evidence that summer school is more effective, on average, for low income students 
compared to higher income students, it also raises questions that the current body of evidence 
cannot answer conclusively. 
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Table 1. 


Meta-analytic Effects for Math Programs and Moderator Analysis: Student Income 
Status 


Analysis 

k 

d 

95% Cl 

Low High 

estimate estimate 

Q, 

T within 

Q be tween 

(p-value) 

Income Moderator 
Math (All studies) 

12 

0.080 

0.023 

0.137 

26.22 ** 

58.00% 

2.802 

(0.094) 

Math (low income) 

8 

0.110 

0.039 

0.183 

18.52 ** 

62.20% 


Math (mixed income) 

3 

-0.017 

-0.237 

0.203 

5.14- 

61.10% 



Note: k=number of studies. Models are random effects models. One math study in the overall results did not report information on student 
income status and is therefore excluded from income status moderator analysis 


SREE Spring 2014 Conference Abstract Template 


AS 



Table 2. 


Within-study Comparisons of Low-income Vs. Middle-income Students (Math and Reading) and Math Vs. 
Reading Effects 


Comparison 



Fixed Effect Model 


Low-income v. middle-income students, math 


Low-income 

Mixed-income 

Difference: Low-income - 
Mixed-income 


d 

0.107 

-0.053 

0.16 


95% Cl 

-0.016 to 0.230 

-0.148 to 0.042 

0.004 to 0.317 


k 

3 

3 

3 


Q 

3.1 

1.35 

0.51 


I 2 

35.50% 

0.00% 

0.00% 

Low-income v. middle-income students, 
reading 


Low-income 

Mixed-income 

Difference: Low-income - 
Mixed-income 


d 

0.139 

-0.14 

0.25 


95% Cl 

0.057 to 0.220 

-0.188 to -0.099 

0.15 to 0.35 


k 

7 

7 

7 


Q 

6.197 

26.875 

11.688 


I 2 

3.18 

77.675 

48.67% 


Math v. reading 


Reading 

Math 

Difference: Reading - Math 


d 

0.02 

0.051 

-0.024 


95% Cl 

-0.006 to 0.047 

0.028 to 0.074 

-0.059 to 0.011 


k 

9 

9 

9 


Q 

15.37- 

20.18* 

0.83 


I 2 

48.00% 

60.40% 

0.00% 


Note: k=number of studies. 
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Reading Effect Size 

Figure 1. Scatter plot of math effect sizes on reading effect sizes for studies with both outcomes, 
y=x line in overlay. Bubble sizes are weighted by the inverse of the variance of the math effect 
size. 
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